Input device

ABSTRACT

An input device includes an operation panel that is installed inside a vehicle and is operated by an operating part; a CCD camera that is disposed inside the vehicle to image at least a front side of the operation panel; and a control unit that predicts an action of the operating part based on image information of the CCD camera and performs operation assistance on the operation panel.

CLAIM OF PRIORITY

This application contains subject matter related to and claims thebenefit of Japanese Patent Application No. 2012-205498 filed on Sep. 19,2012, the entire contents of which is incorporated herein by reference.

BACKGROUND OF THE DISCLOSURE

1. Field of the Disclosure

The present disclosure relates to an input device that performsoperation assistance on an operation panel installed in a vehicle.

2. Description of the Related Art

Japanese Unexamined Patent Application Publication No. 2005-274409discloses a car navigation device. The car navigation device disclosedin Japanese Unexamined Patent Application Publication No. 2005-274409includes image determination means configured to identify whether anoperator of an input device is a driver or a passenger on a frontpassenger seat using a camera installed in a vehicle or based on aphotographed image of the camera. When it is identified that theoperator is the driver while the vehicle travels, control is performedsuch that an operation is invalidated.

According to Japanese Unexamined Patent Application Publication No.2005-274409, when an arm is shown in the photographed image, whether theoperator is the driver or the passenger on the front passenger seat isidentified based on the shape of the like of an arm region.

According to Japanese Unexamined Patent Application Publication No.2005-274409, whether a key input from an operation panel is detected,and whether the operator is the driver or the passenger on the frontpassenger seat is determined based on the shape or the like of the armregion shown in a camera image by using the key input as a trigger.

In Japanese Unexamined Patent Application Publication No. 2005-274409,operability on the operation panel is not changed from the related art.That is, for example, when the operator is the passenger on the frontpassenger seat, the operator touches the operation panel to perform aninput, as in the related art. Therefore, satisfactory operability orfast operability may not be obtained, compared to the related art.

In Japanese Unexamined Patent Application Publication No. 2005-274409,when the operator is the driver, the control is performed such that theoperation is invalidated using the key input as a trigger. Therefore,determination whether to invalidate the operation may not be made ontime, and there is a concern that a problem may occur in safety.

Further, in Japanese Unexamined Patent Application Publication No.2005-274409, since the key input first has to be performed to invalidatean operation, the operation is unnecessarily performed.

These and other drawbacks exist.

SUMMARY OF THE DISCLOSURE

Embodiments of the present disclosure provide an input device capable ofimproving operability on an operational panel or safety while a vehicletravels, compared to the related art, by performing operation assistanceon the operation panel based on action prediction of an operating part.

According to an exemplary embodiment, an input device includes: anoperation panel that is installed inside a vehicle and is operated by anoperating part; an imaging element that is disposed inside the vehicleto image at least a front side of the operation panel; and a controlunit that predicts an action of the operating part based on imageinformation of the imaging element and performs operation assistance onthe operation panel.

Thus, the input device includes the control unit capable of predictingan action of the operating part on the operation panel based on theimage information imaged by the imaging element. And, the operationassistance on the operation panel can be performed based on the actionprediction of the operating part. Accordingly, since the operationassistance can be performed at a front position at which an inputoperation is performed on the operation panel, it is possible to obtainquick operability, comfortable operability, and operability differentfrom that of the related art.

Since the input device according to various embodiments is installedinside a vehicle, the above-described fast operability and comfortableoperability, the operation restriction, or the like can result in animprovement in the safety.

According to an exemplary embodiment, the action prediction of theoperating part is performed, and the input operation control isperformed based on the action prediction. The input operation control isnot performed using a key input as a trigger, as in Japanese UnexaminedPatent Application Publication No. 2005-274409, and an unnecessaryaction can be omitted compared to the related art, which also results inan improvement in the above-described safety.

According to an exemplary embodiment, an operation position at which theaction is predicted on the operation panel may be highlighted anddisplayed by the operation assistance of the control unit.

In this case, a display part may be displayed on the operation panel andthe display part located at the operation position may be highlightedand displayed by the operation assistance of the control unit.

According to an exemplary embodiment, the plurality of display parts maybe arranged on the operation panel. When the operating part approachesone of the display parts, the action may be predicted and the displaypart which the operating part approaches may be configured to behighlighted and displayed. At this time, by arranging the plurality ofdisplay parts in a lateral direction perpendicular to a height directionof the operation panel, the action prediction of the operating part canbe performed easily. Thus, the operation assistance can be performedwith high accuracy.

According to an exemplary embodiment, the control unit may specify anaction detection region divided into a plurality of partitions based onthe image information, predict the action based on entrance of theoperating part into a predetermined partition located on a side of theoperation panel in the action detection region, and perform theoperation assistance. Thus, control can be performed easily such thatthe timing at which the operation assistance is performed issubstantially uniform. The load on the control unit performing theoperation assistance can be reduced by performing the operationassistance based on the entrance of the operating part into thepartition close to the operation panel. Thus, it is possible to improvethe accuracy with which the operation assistance is performed.

According to an exemplary embodiment, the control unit may be able toidentify whether an operator on the operation panel is a driver or apassenger other than the driver based on an entrance position of theoperating part into the action detection region, and perform controlsuch that the operation assistance on the operation panel differsaccording to the operator. In this case, the control unit may performcontrol such that the operation assistance is performed only when thecontrol unit identifies that the operator is a passenger on a frontpassenger seat. The control unit may restrict an input operation on theoperation panel more when identifying that the operator is the driverthan when the operator is the passenger on the front passenger seat. Thecontrol unit performs the operation assistance by preferring the actionprediction of the passenger on the front passenger seat, when thecontrol unit identifies that the operator is either of the driver andthe passenger on the front passenger seat.

According to an exemplary embodiment, the control unit may be able toidentify a height position of the operating part, and thus may predictthe action on the operation panel of the operating part based on theheight position of the operating part and perform the operationassistance.

According to the aspect of the invention, the action may be predictedbased on vector information regarding the operating part. Thus, theaction prediction can be performed easily and smoothly.

According to an exemplary embodiment, the action may be predicted byobtaining coordinates of a contour of a hand which is the operatingpart, calculating a position of a finger from the coordinates, andtracking a movement trajectory of the finger. By using the movementtrajectory of the finger, it is possible to perform the more detailedaction prediction.

According to an exemplary embodiment, the operation assistance on theoperation panel can be performed based on the action prediction of theoperating part. Accordingly, since the operation assistance on theoperation panel can be performed at a front position at which an inputoperation is performed on the operation panel, it is possible to obtainoperability different from that of the related art, quick operability,and comfortable operability.

Since the input device is installed inside a vehicle, theabove-described fast operability and comfortable operability, theoperation restriction, or the like can result in the improvement in thesafety.

According to various embodiments, the action prediction of the operatingpart is configured to be performed. The input operation control is notperformed using a key input as a trigger, as in Japanese UnexaminedPatent Application Publication No. 2005-274409, and an unnecessaryaction can be omitted compared to the related art, which also results inan improvement in the above-described safety.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a partial schematic diagram illustrating the inside of avehicle equipped with an input device according to an exemplaryembodiment;

FIG. 2 is a block diagram illustrating the input device according to anexemplary embodiment;

FIG. 3 is a schematic diagram illustrating an image photographed by aCCD camera (imaging element) according to an exemplary embodiment;

FIG. 4A is a schematic diagram illustrating an imaging element, anoperation panel, and an image range photographed by the imaging element,when viewed from a lateral side according to an exemplary embodiment;

FIG. 4B is a schematic diagram illustrating the imaging element, theoperation panel, and the image range photographed by the imagingelement, when viewed from a front side according to an exemplaryembodiment;

FIGS. 5A to 5D are schematic diagrams illustrating a step of estimatinga part of a hand according to an exemplary embodiment;

FIG. 6A is a flowchart illustrating steps from acquisition of imageinformation of the CCD camera (imaging element) to execution ofoperation assistance on the operation panel according to an exemplaryembodiment;

FIG. 6B is a flowchart illustrating the step of estimating particularlya part of the hand according to an exemplary embodiment;

FIG. 7 is a schematic diagram illustrating movement trajectory of anoperating part (hand) of a driver in an action detection regionspecified by the image information of the CCD camera according to anexemplary embodiment;

FIG. 8 is a schematic diagram illustrating a case in which the operatingpart enters a first partition close to the operation panel, when themovement trajectory of the operating part (hand) illustrated in FIG. 7is tracked according to an exemplary embodiment;

FIG. 9 is a schematic diagram illustrating a case in which an operatingpart (hand) of a driver directly enters the first partition close to theoperation panel according to an exemplary embodiment;

FIG. 10 is a schematic diagram illustrating an input operation surfaceof the operation panel according to an exemplary embodiment;

FIG. 11A is a schematic diagram illustrating a form of the operationassistance on the operation panel and an expanded display state of anicon scheduled to be subjected to an input operation of the operatingpart based on operation prediction of the operating part according to anexemplary embodiment;

FIG. 11B is a schematic diagram illustrating an expanded display stateof the icon as a form different from that of FIG. 11A according to amodification example of FIG. 11A according to an exemplary embodiment;

FIG. 12 is a schematic diagram illustrating a form of the operationassistance on the operation panel and a lighting state of an iconscheduled to be subjected to an input operation of the operating partbased on operation prediction of the operating part according to anexemplary embodiment;

FIG. 13 is a schematic diagram illustrating a form of the operationassistance on the operation panel and a state in which a cursor isdisplayed in an overlapping manner on an icon scheduled to be subjectedto an input operation of the operating part based on operationprediction of the operating part according to an exemplary embodiment;

FIG. 14 is a schematic diagram illustrating a form of the operationassistance on the operation panel and a state in which icons other thanan icon scheduled to be subjected to an input operation of the operatingpart are displayed in a gray-out manner based on operation prediction ofthe operating part according to an exemplary embodiment;

FIG. 15 is a schematic diagram illustrating a form of the operationassistance on the operation panel and a state in which all of the iconson the operation panel are displayed in a gray-out manner according toan exemplary embodiment;

FIG. 16 is a schematic diagram illustrating movement trajectory of anoperating part (hand) of a passenger (operator) on a front passengerseat within an action detection region specified by image information ofthe CCD camera according to an exemplary embodiment;

FIG. 17 is a schematic diagram illustrating movement trajectory of anoperating part (hand) of a passenger (operator) on a rear seat within anaction detection region specified by image information of the CCD cameraaccording to an exemplary embodiment;

FIG. 18 is a schematic diagram illustrating an example in which movementtrajectory of an operating part (hand) of a driver different from thedriver of FIG. 8 is tracked according to an exemplary embodiment;

FIG. 19 is a schematic diagram illustrating a state in which bothoperating parts (hands) of both a driver and a passenger on the frontpassenger seat enter an action detection region according to anexemplary embodiment; and

FIG. 20 is a schematic diagram illustrating an algorithm configure totrack the position of a finger according to an exemplary embodiment.

DETAILED DESCRIPTION OF THE DISCLOSURE

The following description is intended to convey a thorough understandingof the embodiments described by providing a number of specificembodiments and details involving an input device. It should beappreciated, however, that the present invention is not limited to thesespecific embodiments and details, which are exemplary only. It isfurther understood that one possessing ordinary skill in the art, inlight of known systems and methods, would appreciate the use of theinvention for its intended purposes and benefits in any number ofalternative embodiments, depending on specific design and other needs.

FIG. 1 is a partial schematic diagram illustrating the inside of avehicle equipped with an input device according to an embodiment. FIG. 2is a block diagram illustrating the input device according to theembodiment. FIG. 3 is a schematic diagram illustrating an imagephotographed by a CCD camera (imaging element). FIG. 4A is a schematicdiagram illustrating an imaging element, an operation panel, and animage range photographed by the imaging element, when viewed from alateral side. FIG. 4B is a schematic diagram illustrating the imagingelement, the operation panel, and the image range photographed by theimaging element, when viewed from a front side.

FIG. 1 illustrates the vicinity of an in-vehicle anterior row of avehicle. The vehicle in FIG. 1 is a left-hand handle vehicle. However,an input device according to the embodiment is applicable also to aright-hand drive vehicle.

As illustrated in FIG. 1, a CCD camera (imaging element) 11 may bemounted on a ceiling 10 in the vehicle. In FIG. 1, the CCD camera 11 maybe disposed near a rear-view mirror 12. As long as an image photographedby the CCD camera 11 shows at least the front side of the operationpanel 18, the installation position of the CCD camera 11 may not beparticularly limited. The CCD camera 11 may be used, but an action of anoperating part can be detected even during the night when a cameracapable of detecting an infrared ray may be used.

As illustrated in FIG. 1, a central operation unit 17 or an operationpanel 18 including a shift operating part 16 disposed at a positionbetween a driver seat 14 and a front passenger seat 15 may be disposedin a center console 13.

The operation panel 18 may be, for example, an electrostatic capacitancetouch panel and can display a map of a car navigation device, a musicreproduction screen, or the like. An operator is assumed to perform aninput operation with his or her finger or the like directly on a screenof the operation panel 18.

As illustrated in FIG. 4A, the CCD camera 11 may be mounted on theceiling 10 and installed at a position at which at least the front sideof the operation panel 18 is imaged. Here, the front side of theoperation panel 18 may be a direction 18 b perpendicular to a screen 18a of the operation panel 18 and may indicate a spatial region 18 c on aside on which an input operation is performed on the operation panel 18with a finger or the like.

A numeral 11 a illustrated in FIGS. 4A and 4B denotes a central axis(optical axis) of the CCD camera 11 and R denotes an imaging range.

As illustrated in FIG. 4A, when the imaging range R is viewedhorizontally (from the lateral side), the operation panel 18 and thespatial region 18 c located on the front side of the operation panel 18are shown in the imaging range R. As illustrated in FIG. 4B, when theimaging range R is viewed from the front side, a width (the shownlargest width of image information) T1 of the imaging range R is largerthan a width T2 of the operation panel 18.

As illustrated in FIG. 2, an input device 20 according to thisembodiment may include the CCD camera (imaging element) 11, theoperation panel 18, and a control unit 21.

As illustrated in FIG. 2, the control unit 21 may include an imageinformation detection unit 22, a region regulation unit 23, acalculation unit 24, an action prediction unit 25, and an operationassistance function unit 26.

Here, in FIG. 2, one control unit 21 is collectively illustrated.However, for example, a plurality of control units 21 may be provided.The image information detection unit 22, the region regulation unit 23,the calculation unit 24, the action prediction unit 25, and theoperation assistance function unit 26 illustrated in FIG. 2 may bedivided and embedded in the plurality of control units.

That is, how to embed the image information detection unit 22, theregion regulation unit 23, the calculation unit 24, the actionprediction unit 25, and the operation assistance function unit 26 can beappropriately selected.

An action prediction device 28 may be configured to include the CCDcamera (imaging element) 11 and the control unit 29 including the imageinformation detection unit 22, the region regulation unit 23, thecalculation unit 24, and the action prediction unit 25 illustrated inFIG. 2. The action prediction device 28 may be embedded in the vehicleand a vehicle system capable of transmitting and receiving a signal toand from the operation panel 18 is included in the input device 20.

The image information detection unit 22 may acquire image informationimaged by the CCD camera 11. Here, the image information may beelectronic information regarding an image obtained through photography.FIG. 3 is a diagram illustrating a screen 34 imaged by the CCD camera11. As illustrated in FIG. 3, the operation panel 18 and the spatialregion 18 c on the front side of the operation panel 18 are shown on thescreen 34. The central operation unit 17 in which the shift operatingpart 16 and the like are disposed is shown on the front side of theoperation panel 18. Both left and right regions 35 and 36 of theoperation panel 18 and the central operation unit 17 are also shown onthe screen 34 of FIG. 3. The left region 35 may be a driver seat sideregion and the right region 36 may be a front passenger seat sideregion. In FIG. 3, images shown on both the left and right regions 35and 36 are not illustrated. A kind of CCD camera 11, the number ofpixels, or the like is not particularly limited.

The region regulation unit 23 illustrated in FIG. 2 may specify a regionto be used to track a movement trajectory of an operating part and makean action prediction of the operating part based on the imageinformation acquired from the CCD camera 11.

The region regulation unit 23 may specify an image center region locatedon the front side of the operation panel 18 as an action detectionregion 30 on the screen 34 illustrated in FIG. 3. That is, the actiondetection region 30 may be a region surrounded by a plurality of sides30 a to 30 d, and both left and right regions 35 and 36 may be separatedby the action detection region 30. The boundary lines (sides) 30 a and30 b between the action detection region 30 and both left and rightregions 35 and 36 illustrated in FIG. 3 may be indicated by dottedlines. In FIG. 3, the sides 30 c and 30 d may be set as end portions inthe front and rear directions of the screen 34. However, the sides 30 cand 30 d may be disposed inside the screen 34.

The entire screen 34 illustrated in FIG. 3 may be set as the actiondetection region 30. However, in this case, an amount of calculationconsumed for the tracking of the movement trajectory or the actionprediction of an operating part may increase, thereby causing delay ofthe action prediction or short life span of the device. Further,production cost may increase to enable a large amount of calculation.Accordingly, the entire screen 34 may not be used and it may be suitableto use a restricted range as the action detection region 30.

In the form illustrated in FIG. 3, the action detection region 30 may bedivided into two partitions 31 and 32. A boundary line 33 between thepartitions 31 and 32 is indicated by a one-dot chain line. When theaction detection region 30 is divided into a plurality of partitions,how to divide the action detection region 30 can be arbitrarilydetermined. The action detection region 30 may be divided into thenumber of partitions greater than 2. Further, the partition 31 can belocated closer to the operation panel 18. Since an action state of anoperating part in the partition 31 is important to the action predictionof the operating part and the execution of the operation assistance ofthe operation panel 18, an execution timing of the operation assistancecan finely be determined by dividing the inside of the partition 31 inmore detail.

Hereinafter, the partition 31 is referred to as a first partition andthe partition 32 is referred to as a second partition. As illustrated inFIG. 3, the first partition 31 may include the operation panel 18 insidethe image and may be a region closer to the operation panel 18 than thesecond partition 32.

The calculation unit 24 illustrated in FIG. 2 may be a unit thatcalculates movement trajectory of an operating part within the actiondetection region 30. The calculation method may not be particularlylimited. However, for example, the movement trajectory of the operatingpart can be calculated according to the following method.

In FIG. 5A, information regarding a contour 42 of an arm 40 and a hand41 may be detected. When the contour 42 is comprehended, the size of animage captured by the CCD camera 11 may be decreased to reduce theamount of calculation, and then a process of converting the image into amonochrome image may be performed to perform a recognition process. Atthis time, the recognition of the operating part can be performed withhigh accuracy using the detailed image. In an exemplary embodiment,however, the amount of calculation may be reduced by decreasing the sizeof the image so that a quick process can be performed. Thereafter, afterthe image may be converted into the monochrome image, the operating partmay be detected based on a change in luminance. When an infrareddetection camera is used, it may not be necessary to perform the processof converting the image into the monochrome image. Thereafter, forexample, a motion vector may be detected by calculating an optical flowusing a previous frame and a current frame. At this time, the motionvector may be averaged to 2×2 pixels in order to reduce the influence ofnoise. When the motion vector has a length equal to or greater than apredetermined vector length (amount of movement), as illustrated in FIG.5A, the contour 42 from the arm 40 to the hand 41 shown within theaction detection region 30 is detected as an operating part.

Next, as illustrated in FIG. 5A, a longitudinal length (Y1-Y2) of theimage may be restricted. As illustrated in FIG. 5B, an image may be cutout to estimate a region of the hand 41. At this time, the size of eachportion of the operating part may be calculated from the contour 42 anda region with a value equal to or greater than a determined value is setas an effective region. The reason for setting the lower limit is thatthe arm may be excluded using the fact that the width of a hand isgenerally larger than that of an arm. Further, the reason for notsetting the upper limit is that a motion vector may be produced in aconsiderable area when a body part is also imaged within the actiondetection region 30, and thus the contour of the body part may not bedetected in some cases when the upper limit is set. Then, a regioncircumscribed by the contour 42 may be detected within the effectiveregion. For example, in FIG. 5B, the XY coordinates forming the entirecontour 42 may be investigated, the minimum and maximum values of the Xcoordinates may be calculated, and the width (the length in the Xdirection) of the effective region may be reduced, as illustrated inFIG. 5C. By doing so, a minimum rectangular region 43 circumscribed bythe contour 42 may be detected and it may be determined whether thelongitudinal length (Y1-Y2) of the minimum rectangular region 43(effective region) is equal to or less than a predetermined thresholdvalue. When the longitudinal length of the minimum rectangular region 43is equal to or less than the predetermined threshold value, a centroid Gof the effective region may be calculated.

Conversely, when the longitudinal length (Y1-Y2) of the minimumrectangular region 43 (effective region) is equal to or greater than thepredetermined threshold value, a longitudinal length which is the lowerlimit size of the arm is restricted from the Y1 side within apredetermined length range from the Y1 side and an image may be cut out(FIG. 5D). A minimum rectangular region 44 circumscribed by the contour42 may be detected in the more cut-out image, and a region obtained byenlarging the minimum rectangular region 44 by a few pixels in all ofthe directions may be set as a hand estimation region. By setting theenlarged region as the hand estimation region, the region of the hand 41excluded unintentionally during the process of detecting the contour 42can be recognized again. The above-described process of determining theeffective region is performed again in the hand estimation region. Whenthe longitudinal length of an effective region is equal to or less thana predetermined threshold value, the center of the effective region maybe determined as the centroid G of the hand 41. The method ofcalculating the centroid G is not limited to the above-described method,but the centroid G may be also calculated according to an algorithmpresent from the related art. However, due to the action prediction ofan operating part performed while a vehicle travels, quick calculationof the centroid G may be necessary and very high accuracy of theposition of the calculated centroid G may not be necessary. Inparticular, it is important to continuously calculate a motion vector ata position determined as the centroid G. By using this motion vector,the action prediction can reliably be performed even in a case it isdifficult to comprehend the shape of a hand which is an operating part,for example, in a circumstance in which a state of the surroundingillumination is successively changed. In the above-described process,the hand and the arm can reliably be distinguished from each other usingtwo pieces of the information regarding the contour 42 and theinformation regarding the region circumscribed by the contour 42.

During the detection of the above-described motion vector, the motionvector of the centroid G of a moving part (herein, the hand 41) can becalculated and the motion vector of the centroid G can be obtained asthe movement trajectory of the moving part.

Based on the movement trajectory of the operating part, the actionprediction unit 25 illustrated in FIG. 2 predicts subsequent positionwhich the operating part reaches. For example, the action predictionunit 25 predicts a position which the operating part reaches on thescreen 18 a of the operation panel 18, when the operating part continuesto move, depending on whether the movement trajectory of the operatingpart is straight toward the operating panel 18 or the movementtrajectory is obliquely inclined with respect to the operation panel 18.

The operation assistance function unit 26 illustrated in FIG. 2 mayperform operation assistance on the operation panel 18 based on theaction prediction of the operating part. In this embodiment, the“operation assistance” refers to control and adjustment of an inputoperation or a display form or the like of an input operation positionso that satisfactory operability or high safety can be ensured. Aspecific example of the operation assistance will be described below.

Hereinafter, steps from acquisition of image information to execution ofthe operation assistance will be described with reference to theflowchart of FIG. 6A.

First, in step ST1 illustrated in FIG. 6A, the image information of theCCD camera 11 may be acquired by the image information detection unit 22illustrated in FIG. 2. In step ST2, the region regulation unit 23illustrated in FIG. 2 may specify the action detection region 30 fromthe image information and also may divide the inside of the actiondetection region 30 into the plurality of partitions 31 and (see FIGS.5A to 5D).

The entire screen 34 illustrated in FIG. 3 may also be determined as theaction detection region 30. However, at least a region on the front sideof the operation panel 18 may be specified as the action detectionregion 30 in order to reduce the amount of computation (amount ofcalculation).

Subsequently, in step ST3 illustrated in FIG. 6A, the calculation unit24 illustrated in FIG. 2 may perform detection of the motion vector. Thedetection of the motion vector is illustrated in only step ST3illustrated in FIG. 6A, but whether the motion vector is present isnormally detected between a previous frame and a current frame.

In step ST4 illustrated in FIG. 6A, an operating part (hand) may bespecified, as illustrated in FIGS. 5A to 5D, and the calculation unit 24illustrated in FIG. 2 may calculate the centroid G of the operating part(hand).

In an exemplary embodiment, a hand part may be used as the operatingpart, as illustrated in FIGS. 5A to 5D. A flowchart up to the estimationof the hand part and the calculation of the centroid G of the hand isillustrated in FIG. 6B.

In FIG. 6B, the image may be acquired by the CCD camera 11, asillustrated in FIG. 6A. Thereafter, the size of the image may be reducedin step ST10, and then the process of converting the image into amonochrome image may be performed in step ST11 to perform therecognition process. Subsequently, in step ST12, for example, the motionvector may be detected by calculating the optical flow using a previousframe and a current frame. The detection of the motion vector is alsoillustrated in step ST3 of FIG. 6A. In FIG. 6B, when the motion vectoris detected, the process proceeds to subsequent step ST13.

In step ST13, the motion vector may be averaged to 2×2 pixels. Forexample, 80×60 blocks are obtained at this time.

Next, in step ST14, a vector length (amount of movement) may becalculated for each block. When the vector length is greater than adetermined value, the block is determined as a block having effectivemotion.

Subsequently, as illustrated in FIG. 5A, the contour 42 of the operatingpart may be detected (step ST15).

Next, in step ST16, the size of each portion of the operating part maybe calculated from the contour 42 and the region with a value equal toor greater than the determined value is set as an effective region. Aregion circumscribed by the contour 42 may be detected within theeffective region. As described with reference to FIG. 5B, for example,the XY coordinates forming the entire contour 42 may be investigated,the minimum and maximum values of the X coordinates are calculated, andthe width (the length in the X direction) of the effective region may bereduced, as illustrated in FIG. 5C.

By doing so, the minimum rectangular region 43 circumscribed by thecontour 42 may be detected. In step ST17, it may be determined whetherthe longitudinal length (Y1-Y2) of the minimum rectangular region 43(effective region) may be equal to or less than the predeterminedthreshold value. When the longitudinal length of the minimum rectangularregion 43 is equal to or less than the predetermined threshold value,the centroid G within the effective region may be calculated asillustrated in step ST18.

Conversely, when the longitudinal length (Y1-Y2) of the minimumrectangular region 43 (effective region) is equal to or greater than thepredetermined threshold value in step ST17, the longitudinal lengthwhich is the lower limit size of the arm may be restricted within thepredetermined length range from the Y1 side and an image is cut out (seeFIG. 5D). Then, as illustrated in step ST19, the minimum rectangularregion 43 circumscribed by the contour 42 may be detected in the cut-outimage, and the region obtained by enlarging the minimum rectangularregion 43 by a few pixels in all of the directions is set as the handestimation region.

In step ST20 to step ST22, the same steps as step ST14 to step ST16 maybe performed in the above-described hand estimation region. Thereafter,in step ST19, the center of the effective region may be determined asthe centroid G of the hand 41.

As described above, the centroid G of the operating part (hand) may becalculated. Thereafter, in step ST5 illustrated in FIG. 6A, the movementtrajectory of the operating part (hand) may be tracked. Here, thetracking of the movement trajectory can be obtained by the motion vectorof the centroid G. The tracking refers to a state in which a motion ofthe hand entering the action detection region 30 may be continuouslytracked. As described above, the movement trajectory can be tracked bythe movement vector of the centroid G of the hand. However, theacquisition of the centroid G may be performed, for example, when themotion vector is detected by calculating the optical flow using aprevious frame and a current frame. There is a temporal interval betweenthe acquisitions of the centroid G. However, in this embodiment, thetracking may be performed in addition to the temporal interval betweenthe acquisitions of the centroid G.

The tracking of the movement trajectory of the operating part may startwhen it is detected that the operating part enters the action detectionregion 30. After a while, for example, the tracking of the movementtrajectory of the operating part may start after it is determined thatthe operating part reaches the vicinity of the boundary line 33 betweenthe first partition 31 and the second partition 32. The start time ofthe tracking the movement trajectory can be arbitrarily determined. Inan exemplary embodiment, the tracking of the movement trajectory may beassumed to start when it is determined that the operating part entersthe action detection region 30.

FIG. 7 illustrates a state in which a driver is now stretching his orher hand 41 in the direction of the operation panel 18 to operate theoperation panel 18.

An arrow L1 illustrated in FIG. 7 indicates a movement trajectory(hereinafter, referred to as a movement trajectory L1) of the hand 41within the action detection region 30.

As illustrated in FIG. 7, the movement trajectory L1 of the hand 41 isbeing moved in the direction of the first partition 31 within the secondpartition 32 more distant from the operation panel 18 between theplurality of partitions 31 and 32 of the action detection region 30.

In step ST6 illustrated in FIG. 6A, it may be detected whether themovement trajectory L1 enters the inside of the first partition 31closer to the operation panel 18. When the movement trajectory L1 doesnot enter the inside of the first partition 31, the process may returnto step ST5 and the movement trajectory L1 of the hand 41 may becontinuously tracked according to the routine of step ST3 to step ST5illustrated in FIG. 6A. Thus, although not illustrated in FIG. 6A, theroutine of step ST3 to step ST15 may be normally operated during theaction prediction even after the process returns to step ST5.

As illustrated in FIG. 8, when the movement trajectory L1 of the hand 41enters the inside of the first partition 31 closer to the operationpanel 18 from the second partition 32, step ST6 illustrated in FIG. 6Ais satisfied and the process proceeds to step ST7. Whether the movementtrajectory L1 enters the inside of the first partition 31 can bedetected by the calculation unit 24 illustrated in FIG. 2. Also, apartfrom the calculation unit 24, the control unit 21 may include adetermination unit that determines whether the movement trajectory L1enters the first partition 31.

In step ST7 illustrated in FIG. 6A, the action prediction of the hand(operating part) 41 may be performed based on the movement trajectoryL1. That is, when the movement trajectory is not changed and themovement trajectory L1 arriving from the second partition 32 to thefirst partition 31 may be maintained, the action prediction unit 25illustrated in FIG. 2 may predict a position at which the hand 41reaches the action detection region 30 (a side of the screen 18 a of theoperation panel 18 which the hand 41 reaches). By further dividing thepartition may be further divided according to the position of anoperating member such as the shift operating part 16 present within theaction detection region 30, various measures can be taken when it ispredicted that the shift operating part 16 is about to be operated. Forexample, the shift operating part 16 can be illuminated by separatelyinstalled illumination means.

In FIG. 8, the movement trajectory L1 of the hand 41 may be moved fromthe second partition 32 of the action detection region 30 to the firstpartition 31. However, for example, as illustrated in FIG. 9, a movementtrajectory L2 of the hand 41 may directly enter the first partition 31without passing through the second partition 32 of the action detectionregion 30.

FIG. 10 illustrates the screen 18 a of the operation panel 18. Asillustrated in FIG. 10, a plurality of icons A1 to A8 may be arranged ina lateral direction (X1-X2) perpendicular to a height direction (Z1-Z2)of the operation panel 18 in the lower side of the screen 18 a. Aportion above each of the icons A1 to A8 is a portion in which a map maybe displayed or music reproduction may be displayed in the carnavigation device.

Unlike the arrangement of the icons A1 to A8 illustrated in FIG. 10, forexample, a configuration may be realized in which the icons A1 to A8 arearranged in the height direction (Z1-Z2). Also, for example, aconfiguration may be realized in which some of the icons are arranged inthe lateral direction and the remaining icons are arranged in the heightdirection.

However, in the configuration in which the icons are arranged in theheight direction, when the movement trajectory L1 or L2 of the hand 41enters the first partition 31 as in FIG. 8 or 9 or when the movementtrajectory L1 is located inside the second partition 32 as in FIG. 7, itmay be necessary to detect a height position of the hand 41. Here, themethod of calculating the height position of the operating part may notbe limited. However, for example, the height position of the hand 41 canbe estimated based on the sizes of the minimum rectangular regions 43and 44 in which the contour 42 of the hand 41 is entered in FIGS. 5C and5D. That is, as illustrated in FIG. 3, the screen 34 shown by the CCDcamera 11 may be planar and only the planar information can be obtained.Therefore, in order to know the height position of the hand 41, the hand41 can be detected at a higher position (closer to the CCD camera 11) asthe areas of the minimum rectangular regions 43 and 44 may be larger. Atthis time, in order to calculate the height position of the hand 41 byan area change with respect to a reference size (for example, the sizeof the hand 41 when the center of the operation panel 18 is operated) ofthe hand 41, initial setting may be performed to measure the size of thereference. Thus, it may be possible to estimate a height position atwhich the movement trajectory of the hand 41 is present.

Now, assume that an input operation on the icon A1 illustrated in FIG.10 is predicted based on the movement trajectory of the hand 41(operating part). Then, the action prediction information may betransmitted to the operation assistance function unit 26 and an operatormay be confirmed in step ST8 illustrated in FIG. 6A. Thereafter, asillustrated in step ST9 of FIG. 6A, the operation assistance on theoperation panel 18 may be performed. For example, as illustrated in FIG.11A, the icon A predicted to be subjected to the input operation isdisplayed in an enlarged manner before a finger comes into contact withthe screen 18 a. This is one display form in which the icon A1 predictedto be subjected to the input operation through the action prediction ishighlighted.

As in FIG. 11B, the icons A1 and A3 located in the vicinity (both sidesof the icon A2) of the icon A2 can be displayed in an enlarged mannertogether with the icon A2 and the remaining icons A4 to A8 can also beerased from the screen, when an input operation on the icon A2illustrated in FIG. 10 is predicted based on a movement trajectory ofthe hand 41 (operating part). Thus, in the configuration in which only aplurality of icons adjacent to each other centering an action predicteddestination is enlarged, the icons can be more enlarged and displayed,and thus an erroneous operation can be prevented. In particular, bydisplaying and enlarging only portions of a plurality of icons predictedto be subjected to an input operation by a driver while a vehicletravels, an operation mistake, such as a case in which the drivererroneously presses down a nearby icon, can be prevented even when thevehicle is shaken.

In an exemplary embodiment, besides FIGS. 11A and 11B, the icon A1 maybe lighted or blinked, as illustrated in FIG. 12. Also, a cursor display50 or another display may overlap the icon A1 so that it is indicatedthat the icon A1 is selected, as illustrated in FIG. 13. The icons A2 toA8 other than the icon A1 also may be displayed in a gray-out manner sothat the fact that only the icon A1 can be input is highlighted anddisplayed, as illustrated in FIG. 14.

As illustrated in FIG. 6A, in step ST8, the operator may be confirmed.However, for example, when it is identified that the operator is thedriver, all of the icons A1 to A8 on the screen 18 a of the operationpanel 18 may be displayed in the gray-out manner as one form of theoperation assistance in order to improve safety while the vehicletravels, as illustrated in FIG. 15. In the form illustrated in FIG. 15,for example, when a traveling speed of the vehicle is calculated from avehicle speed sensor (not illustrated) and the traveling speed is equalto or greater than a predetermined speed and it is recognized that theoperator is the driver, control can be performed such that all of theicons A1 to A8 can be displayed in the gray-out manner, as illustratedin FIG. 15.

The control unit 21 can easily and appropriately determine whether theoperator is the driver or a passenger other than the driver by thetracking of the movement trajectory L1 from the entrance position on theboundary lines (sides) 30 a and 30 b between the action detection region30 and both left and right regions 35 and 36.

That is, as illustrated in FIG. 7, it is identified that the hand 41 isthe hand of the driver (due to the left-hand handle in the formillustrated in FIG. 1) by detecting that the hand 41 enters the insideof the action detection region 30 from the boundary line 30 a betweenthe action detection region 30 and the left region 35 on the driver seatside.

As illustrated in FIG. 16, it can be identified that a hand 60 is thehand of a passenger on the front passenger seat side, when a movementtrajectory L4 of the hand 60 extends from the boundary line 30 b betweenthe action detection region 30 and the right region 36 on the frontpassenger seat to the inside of the action detection region 30.

Also, as illustrated in FIG. 17, it can be identified that the operatoris a passenger on a rear seat, when a movement trajectory L5 enters fromthe position of the side 30 d remotest from the operation panel 18 inthe action detection region 30 to the inside of the action detectionregion 30.

In an exemplary embodiment, by tracking a movement trajectory of anoperating part, that is, by tracking a movement trajectory L6 of a hand41 (operating part), as illustrated in FIG. 18, it can be identifiedthat the operator is the driver even when the driver attempts to operatethe operation panel 18 by turning his or her arm to the front passengerseat side, as illustrated in FIG. 18.

In an exemplary embodiment, the input operation function may bedifferently controlled depending on whether the operator is the driveror a passenger other than the driver. For example, when a passenger onthe front passenger seat is the operator, the control may be performedsuch that the icon A1 illustrated in FIGS. 11A to 14 is highlighted anddisplayed. When the driver is the operator, the control may be performedsuch that all of the icons A1 to A8 illustrated in FIG. 15 are displayedin the gray-out manner. Thus, the safety while the vehicle travels canbe improved. When it is identified that a passenger on a rear seat isthe operator, the safety can be improved, for example, by displaying allof the icons A1 to A8 in the gray-out manner, as in the case in whichthe driver is the operator. Thus, the operation position of theoperation panel 18 may be highlighted and displayed only when it isdetermined that the operator is the passenger on the front passengerseat.

When it is identified in step ST8 illustrated in FIG. 6A that theoperator is the driver, the restriction on the input operation comparedto the case in which the operator is the passenger on the frontpassenger seat is suitable for improving the safety. For example, asdescribed above, when the vehicle is traveling at a speed equal to orgreater than a predetermined speed, control can be considered to beperformed such that the input operation is invalidated by displaying allof the icons in the gray-out manner.

As illustrated in FIGS. 11A and 11B, even when the icon A1 is enlargedand displayed, comfortable operability and the safety can be improved byfurther enlarging and displaying the icon A1 in the case in which thedrive is the operator than the case in which the passenger on the frontpassenger seat is the operator. This case is also an example in whichthe control is performed such that the input operation function differsdepending on whether the operator is the driver or a passenger otherthan the driver.

As illustrated in FIG. 19, when both of a movement trajectory L7 of thehand 41 of the driver and a movement trajectory L8 of the hand 60 of thepassenger on the front passenger seat are detected within the firstpartition 31 of the action detection region 30, a method of performingthe operation assistance by preferring the action prediction of thepassenger on the front passenger seat may be suitable for improving thesafety while the vehicle travels.

The operation assistance on the operation panel 18 also may include, forexample, a form in which an input is automatically turned on or off evenwhen the operation panel 18 is not touched based on action prediction ofan operating part.

As illustrated in FIGS. 11A to 14, when the icon A1 predicted to besubjected to an input operation is highlighted and displayed and thenthe hand 41 further approaches the operation panel 18, the inputoperation on the icon A1 can be confirmed before the finger is touchedon the icon A1.

In an exemplary embodiment, the example has been given in which the iconis highlighted and displayed. However, a display part other than theicon also may be used. For example, a predicted operation position maybe highlighted and displayed.

FIG. 20 is a diagram illustrating a method of detecting a finger. First,the coordinates of the contour 42 of the hand 41 in FIG. 5B may beobtained, and points B1 to B5 located most closely in the Y1 directionmay be listed up, as illustrated in FIG. 20. Since the Y1 directionindicates the direction of the operation panel 18, the points B1 to B5located most closely in the Y1 direction may be predicted to be a frontend of the finger. Of the points B1 to B5, the point B1 located mostclosely to the X1 side and the point B5 located to be the most close tothe X2 side may be obtained. Then, the coordinates (here, the positionof the point B3) of the middle of the points B1 and B5 may be predictedas the position of the finger. In this embodiment, control can also beperformed such that the action prediction is performed by setting afinger as an operating part and tracking the movement trajectory of thefinger. By using the movement trajectory of the finger, more detailedaction prediction can be performed.

Determination of a right hand and a left hand, determination of thefront and back of a hand, or the like may be performed.

Even when an operating object is stopped within the action detectionregion 30, a movement trajectory of the operating part can beimmediately tracked, in spite of the fact that movement of the operatingpart subsequently starts, by frequently acquiring the stop state by acentroid vector or the like or maintaining the centroid G in the stopstate for a predetermined time.

The input device 20 according to an embodiment may include the operationpanel 18 that is installed inside a vehicle and is operated by anoperating part; the CCD camera (imaging element) 11 that is disposedinside the vehicle to image at least the front side of the operationpanel 18; and the control unit 21 that predicts an action of anoperating part on the operation panel 18 based on the image informationof the CCD camera 11 and performs the operation assistance.

In an exemplary embodiment, the input device 20 may include the controlunit 21 capable of predicting an action of the operating part on theoperation panel 18 based on the image information imaged by the CCDcamera 11. The operation assistance on the operation panel 18 can beperformed based on the action prediction of the operating part.Accordingly, since the operation assistance can be performed at a frontposition at which an input operation is performed on the operation panel18, it may be possible to obtain operability different from that of therelated art, quick operability, and comfortable operability.

Since the input device 20 according to an exemplary embodiment may beinstalled inside a vehicle, the safety can be improved due to theabove-described fast operability and comfortable operability, theoperation restriction, or the like.

In an exemplary embodiment, the action prediction of the operating partis performed, the input operation control may be performed based on theaction prediction. The input operation control is not performed using akey input as a trigger, as in Japanese Unexamined Patent ApplicationPublication No. 2005-274409, and an unnecessary action can be omittedcompared to the related art, which also results in an improvement in theabove-described safety.

In an exemplary embodiment, the calculation unit 24 of the control unit21 illustrated in FIG. 2 may track a movement trajectory of an operatingpart. However, the tracking of a movement trajectory is not essential,as long as an action of an operating part can be predicted. For example,when it is determined that an operating part is located, for example, inthe vicinity of the middle of the first partition 31 illustrated in FIG.3, it may be thereafter predicted that the operating part performs aninput operation on the vicinity of the middle of the operation panel 18and the operation assistance such as highlighting display of an icon inthe vicinity of the middle of the operation panel 18 can be performed.In particular, it may be easy to perform the action prediction bydividing the first partition 31 in more detail, and thus the operationassistance on the operation panel 18 can be performed based on theaction prediction.

However, by tracking a movement trajectory of an operating part, it maybe easy to perform the action prediction and the action prediction withhigh accuracy can be performed. Therefore, the action prediction may beperformed based on the tracking of the movement trajectory of theoperating part.

In an exemplary embodiment, an operation position at which the action ispredicted on the operation panel 18 may be highlighted and displayedthrough the operation assistance. For example, a display part typifiedby an icon may be highlighted and displayed. As illustrated in FIG. 10,the icons A1 to A8 may be arranged in the lateral direction (X1-X2) ofthe operation panel 18. Thus, by arranging the icons in the lateraldirection of the operation panel 18, it may not be necessary to detectthe height position of the operating part. Therefore, an action of theoperating part can be predicted easily, and thus the operationassistance can be performed with high accuracy.

In an exemplary embodiment, the control unit 21 may specify the actiondetection region 30 divided into the plurality of partitions 31 and 32to the front side of the operation panel 18 based on the imageinformation of the CCD camera 11, predict the action based on entranceof the operating part into the first partition 31 located on the side ofthe operation panel 18 in the action detection region 30, and performthe operation assistance. Thus, control can be performed easily suchthat the timing at which the operation assistance is performed may besubstantially uniform. The load on the control unit 21 performing theoperation assistance can be reduced by performing the operationassistance based on the entrance of the operating part into the firstpartition 31 close to the operation panel 18. Thus, it may be possibleto improve the accuracy with which the operation assistance isperformed.

In an exemplary embodiment, the control unit 21 may be able to identifywhether the operator on the operation panel 18 is the driver or apassenger other than the driver based on an entrance position of theoperating part into the action detection region 30, and may performcontrol such that the operation assistance on the operation panel 18differs according to the operator. In this case, for example, only whenit is identified that the operator is the passenger on the frontpassenger seat, the control may be performed such that the operationassistance is performed. The input operation on the operation panel 18may be restricted more when it is identified that the operator is thedriver than when the operator is the passenger on the front passengerseat. When it is identified that the operator is either of the driverand the passenger on the front passenger seat, the operation assistancemay be performed by preferring the action prediction of the passenger onthe front passenger seat. Thus, the safety while the vehicle travels canbe improved.

In an exemplary embodiment, the height position of the operating partmay be identified, the action on the operation panel 18 of the operatingpart may be configured to be predicted based on the height position ofthe operating part, and the operation assistance may be performed.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims of the equivalents thereof.

Accordingly, the embodiments of the present inventions are not to belimited in scope by the specific embodiments described herein. Further,although some of the embodiments of the present disclosure have beendescribed herein in the context of a particular implementation in aparticular environment for a particular purpose, those of ordinary skillin the art should recognize that its usefulness is not limited theretoand that the embodiments of the present inventions can be beneficiallyimplemented in any number of environments for any number of purposes.Accordingly, the claims set forth below should be construed in view ofthe full breadth and spirit of the embodiments of the present inventionsas disclosed herein. While the foregoing description includes manydetails and specificities, it is to be understood that these have beenincluded for purposes of explanation only, and are not to be interpretedas limitations of the invention. Many modifications to the embodimentsdescribed above can be made without departing from the spirit and scopeof the invention.

What is claimed is:
 1. An input device comprising: an operation panelinstalled inside a vehicle and capable of being operated by an operatingpart; an imaging element disposed inside the vehicle to image at least afront side of the operation panel; and a control unit that predicts anaction of the operating part based on image information of the imagingelement and performs operation assistance on the operation panel,wherein an operation position at which the action is predicted on theoperation panel is highlighted and displayed by the operation assistanceof the control unit.
 2. The input device according to claim 1, whereindisplay parts are displayed on the operation panel and the display partlocated at the operation position is highlighted and displayed by theoperation assistance of the control unit.
 3. The input device accordingto claim 2, wherein the plurality of display parts are arranged on theoperation panel, and wherein, when the operating part approaches one ofthe display parts, the action is predicted and the display part whichthe operating part approaches is highlighted and displayed.
 4. The inputdevice according to claim 3, wherein the plurality of display parts arearranged in a lateral direction perpendicular to a height direction ofthe operation panel.
 5. The input device according to claim 1, whereinthe control unit specifies an action detection region divided into aplurality of partitions based on the image information, predicts theaction based on entrance of the operating part into a predeterminedpartition located on a side of the operation panel in the actiondetection region, and performs the operation assistance.
 6. The inputdevice according to claim 1, wherein the control unit is able toidentify whether an operator on the operation panel is a driver or apassenger other than the driver based on an entrance position of theoperating part into the action detection region, and performs controlsuch that the operation assistance on the operation panel differsaccording to the operator.
 7. The input device according to claim 6,wherein the control unit performs the operation assistance only when thecontrol unit identifies that the operator is a passenger on a frontpassenger seat.
 8. The input device according to claim 6, wherein thecontrol unit restricts an input operation on the operation panel morewhen identifying that the operator is the driver than when the operatoris the passenger on the front passenger seat.
 9. The input deviceaccording to claim 6, wherein, the control unit performs the operationassistance by preferring the action prediction of the passenger on thefront passenger seat, when the control unit identifies that the operatoris either of the driver and the passenger on the front passenger seat.10. The input device according to claim 1, wherein the control unit isable to identify a height position of the operating part, and thuspredicts the action on the operation panel of the operating part basedon the height position of the operating part and performs the operationassistance.
 11. The input device according to claim 1, wherein theaction is predicted based on vector information regarding the operatingpart.
 12. The input device according to claim 1, wherein the action ispredicted by obtaining coordinates of a contour of a hand which is theoperating part, calculating a position of a finger from the coordinates,and tracking a movement trajectory of the finger.
 13. The input deviceaccording to claim 1, wherein the operation panel includes a screen andan enlarged icon is displayed on the screen.
 14. The input deviceaccording to claim 13, wherein the enlarged icon is displayed on thescreen and surrounding icons are erased from the screen.
 15. The inputdevice according to claim 1, wherein the operation panel includes ascreen and a blinked icon is displayed on the screen.
 16. The inputdevice according to claim 1, wherein the operation panel includes ascreen and a lighted icon is displayed on the screen.
 17. The inputdevice according to claim 1, wherein the operation panel includes ascreen that displays a plurality of icons on the screen and an iconother than an icon to be subjected to action prediction is displayed onthe screen in a grayed-out manner.
 18. The input device according toclaim 17, wherein all of the plurality of icons are displayed on thescreen in a grayed-out manner when a predetermined condition is notsatisfied.